Multisensor Signal Processing: Theory and Algorithms for Image-based Rendering and Multichannel Sampling by Ha Thai Nguyen

Multisensor applications have recently inspired important research projects to utilize existing infrastructure and exploit spatiotemporal information. This dissertation focuses on two multisensor applications: image-based rendering and multichannel sampling. Although many image-based rendering (IBR) algorithms have been proposed, few of them possess rigorous interpolation processes. We propose a conceptual framework, called the Propagation Algorithm, that generalizes many existing IBR algorithms, using calibrated or uncalibrated images, and focuses on rigorous interpolation. We propose novel techniques to remove occlusions, for both calibrated and uncalibrated cases, and to interpolate the virtual image using both intensity and depth. Besides algorithms, quantitative analysis is important to effectively control the quality and cost of IBR systems. We analyze the rendering quality of IBR algorithms using per-pixel depth. Working on the spatial domain, we consider the IBR problem as a nonuniform interpolation problem of the virtual image or the surface texture. The rendering errors can be quantified using the sample errors and jitters. We approximate the actual samples, in the virtual image plane or object surfaces, as a generalized Poisson process, and bound the jitters caused by noisy depth estimates. We derive bounds for the mean absolute error (MAE) for two classes of IBR algorithms: image-space interpolation and objectspace interpolation. The bounds highlight the effects of depth and intensity estimate errors, the scene geometry and texture, the number of actual cameras, their positions and resolution. We find that, in smooth regions, MAE decays as O(λ−2) for 2D scenes and as O(λ−1) for 3D scenes, where λ is the local sample density. Finally, motivated by multichannel sampling applications, we consider hybrid filter banks consisting of fractional delay operators, analog analysis filters, slow A/D converters, digital expanders, and digital synthesis filters to approximate a fast A/D converter. The synthesis filters are to be designed to minimize the maximum gain of an induced error system. We show the equivalence of this system to a digital system, used to design the synthesis filters using control theory tools, including model-matching and linear matrix inequality. The designed system is robust against delay estimate errors.